SYM-AM-15-1 10 


PROCEEDINGS 

OF  THE 

TWELFTH  ANNUAL  ACQUISITION 
RESEARCH  SYMPOSIUM 


THURSDAY  SESSIONS 
VOLUME  II 


Orchestrating  the  Development  of  a  Complex  System  of 
Systems:  Systems  Engineering  Tools  and  Methodologies 

Carly  Jackson,  SPAWAR  Systems  Center  Pacific 
Richard  Volkert,  SPAWAR  Systems  Center  Pacific 
George  Saroch,  PMS  420 

Published  April  30,  2015 


Disclaimer:  The  views  represented  in  this  report  are  those  of  the  author  and  do  not  reflect  the  official  policy 
position  of  the  Navy,  the  Department  of  Defense,  or  the  federal  government. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

30  APR  2015  2' REPORT  TYPE 

3.  DATES  COVERED 

00-00-2015  to  00-00-2015 

4.  TITLE  AND  SUBTITLE 

Orchestrating  the  Development  of  a  Complex  System  of  Systems:  Systems 
Engineering  Tools  and  Methodologies 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

SPA  WAR  Systems  Center  (SSC)  Pacific, 53560  Hull  Street, San 

Diego, CA, 92152 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

ARSTRATT 

1 8 .  NUMBER  1 9a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  aS 

unclassified  unclassified  unclassified  Report  (SAR) 

15 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


The  research  presented  in  this  report  was  supported  by  the  Acquisition  Research 
Program  of  the  Graduate  School  of  Business  &  Public  Policy  at  the  Naval 
Postgraduate  School. 

To  request  defense  acquisition  research,  to  become  a  research  sponsor,  or  to  print 
additional  copies  of  reports,  please  contact  any  of  the  staff  listed  on  the  Acquisition 
Research  Program  website  (www.acquisitionresearch.net). 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Orchestrating  the  Development  of  a  Complex  System  of 
Systems:  Systems  Engineering  Tools  and  Methodologies 


Carly  Jackson — is  a  systems  engineer  with  SPAWAR  Systems  Center  (SSC)  Pacific  and  a 
Distributed  Surveillance  Systems  branch  supervisor.  She  is  currently  supporting  the  LCS  MM 
Program  Technical  Director’s  office,  primarily  tasked  with  the  implementation  of  product  and  process 
commonality  across  complex  systems  of  systems,  systems  engineering  preparations  for  the 
impending  Milestone  B,  technology  transition  planning,  and  overall  management  of  the  Mission 
Package  Technology  Development  Working  Group.  Jackson  earned  simultaneous  BS  and  MS 
degrees  in  mechanical  engineering  from  UCLA  in  2002  and  an  MBA  in  business  administration  from 
Pepperdine  University  in  2007.  She  is  Level  III  certified  by  the  DAU  in  SPRDE-Systems  Engineering 
and  Level  II  certified  in  Program  Management,  [carly.jackson@navy.mil] 

Rich  Volkert — is  the  lead  systems  engineer  supporting  SSC  Pacific,  and  the  PMS  420  deputy 
technical  director.  He  has  over  28  years  of  service  in  the  government,  including  20  years  as  an  active 
duty  naval  officer  with  service  as  a  engineering  duty  officer  and  in  submarines.  Over  19  years  of  that 
time  he  has  been  involved  in  the  fields  of  research,  development,  acquisition,  and  systems 
engineering.  He  possesses  degrees  in  aerospace  engineering  and  acoustical  engineering  and  is 
presently  enrolled  in  a  PhD  program  for  systems  engineering.  He  is  Level  III  certified  by  the  DAU  in 
SPRDE-Systems  Engineering,  Program  Systems  Engineering,  Test  and  Evaluation,  and  Program 
Management,  richard.volkert@navy.mil] 

Introduction 

Acquisition  efforts  are  under  increasing  pressure  to  deliver  systems  rapidly  and  at 
lower  cost  while  providing  enhanced  performance  capability.  This  is  happening  in  an 
environment  where  research  and  development  budgets  are  decreasing  across  the 
Department  of  Defense  (DoD),  the  acquisition  workforce  is  rebuilding,  and  the  DoD  is 
looking  to  rebalance  warfighting  portfolios.  This  has  been  compounded  by  an  extremely 
dynamic  environment  of  fiscal  change  and  uncertainty  imposed  by  significant  budget  cuts 
and  sequestration  impacts  in  the  recent  past. 

The  overall  trend  of  DoD  budgets  has  been  declining.  The  president’s  fiscal  year 
(FY)  2015  budget  recognizes  the  fiscal  imperative  of  deficit  reduction,  reducing  projected 
defense  budgets  by  about  $113  billion  over  five  years  compared  to  levels  requested  in  the 
FY2014  budget  (DoD,  2014).  As  a  percentage  of  Gross  Domestic  Product  (GDP),  the  DoD 
budget  has  been  declining  from  approximately  15%  in  the  mid-1950s  to  around  3%  today. 
The  growth  in  the  DoD’s  costs  under  the  Congressional  Budget  Office  (CBO)  projection  of 
the  base  budget  is  somewhat  less  than  the  CBO’s  (2014)  projection  of  the  growth  of  the 
U.S.  economy,  decreasing  slowly  from  2.8%  of  GDP  in  2015  to  2.5%  in  2025,  and  2.3%  in 
2030. 


One  outcome  of  the  fiscal  pressures  has  been  an  increase  in  defense  acquisition 
efforts  that  are  focused  on  the  rapid  development  and  fielding  of  an  integrated  system  of 
systems  (SoS)  capability.  Many  of  these  SoS  leverage  mature  systems  and  sub-systems  as 
“constituent”  systems  but  do  not  contract  directly  with  industry  to  deliver  the  integrated 
capability.  Elements  of  the  DoD  acquisition  workforce  are  therefore  taking  on  increasingly 
technical  roles  and  responsibilities  as  the  lead  integrators  for  complex  SoS.  It  should  also  be 
noted  that  the  acquisition  workforce  (AWF)  has  declined  by  more  than  50%  since  the  1990s, 
while  the  value  of  Department  of  the  Navy  (DoN)  contracting  has  increased  by  more  than 
50%.  With  these  significant  AWF  losses  and  attendant  workload  increases,  the  DoN 
recognized  the  loss  in  ability  to  manage  the  technical-cost  tradespace  it  is  responsible  to 
execute,  including  major  weapons  systems  acquisition  (DoN,  2010).  While  the  DoN  AWF 
has  grown  by  more  than  20%  (DoN,  2015)  since  2010,  some  program  managers  are  still 
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challenged  with  staffing  and  resourcing  an  acquisition  workforce  with  the  technical  depth  to 
assume  the  increasingly  technical  roles  associated  with  the  development  and  fielding  of  an 
inherently  government  SoS  capability.  In  many  cases,  the  Naval  Warfare  Center  and 
Systems  Center’s  engineering  workforce  is  tasked  to  supplement  the  acquisition  workforce 
with  technical  subject  matter  expertise  in  systems  engineering,  rapid  prototyping, 
development  and  integration,  test  and  evaluation,  operations  analysis,  and  other  inherently 
government  roles. 

Several  novel  systems  engineering  tools  and  methodologies  have  been  developed  to 
support  the  lead  SoS  engineer  (LSoSE)  with  assessing  system  performance,  system 
readiness,  and  risk  through  incremental  development  of  a  complex  SoS.  This  paper  will 
review  the  systems  engineering  processes  and  required  level  of  technical  rigor  to  manage 
the  development  and  integration  of  a  complex  SoS  in  a  distributed  environment.  A  specific 
implementation  of  a  Kill  Chain  framework  that  was  designed  to  provide  the  PM  and  LSoSEs 
with  comprehensive  insight  into  the  technical  status  and  major  risks  for  a  complex  SoS 
across  specific  mission  threads  will  be  presented.  This  paper  will  also  review  some  of  the 
systems  engineering  tools  and  methodologies  developed  and  implemented  by  the  Littoral 
Combat  Ships  (LCS)  Mission  Modules  (MM)  Program  Office  (PMS  420)  that  support  the 
LCS  MM  Kill  Chain  framework. 

LCS  Program  Overview  and  Background 
Acquisition  Strategy 

The  Flight  Zero  Capability  Development  Document  (CDD)  for  the  Littoral  Combat 
Ship  (LCS)  was  approved  in  April  2004  and  established  the  requirements  of  the  LCS  and 
three  “focused”  missions.  Each  Mission  Package  (MP)  provides  warfighting  capability  for 
one  of  three  focused  mission  areas: 

•  Mine  Countermeasures  (MCM) — Detection  and  neutralization  of  mine  threats 

•  Surface  Warfare  (SUW) — Maritime  security  missions  and  fleet  protection  from 
small  boats  and  other  asymmetric  threats 

•  Anti-Submarine  Warfare  (ASW) — Countering  shallow  water  diesel  submarine 
threats 

The  CDD  was  updated  in  2008  to  the  current  Flight  0+  version  which  included 
several  requirement  updates,  which  were  used  to  drive  the  design  for  the  full  “production” 
versions  of  the  seaframe  and  MPs.  MP  specific  requirement  updates  included  the  Net 
Ready  Key  Performance  Parameter  (KPP),  Material  Availability  KPP,  and  others. 

In  2003,  the  LCS  MM  Program  Office  (PMS  420)  was  established  as  part  of  the 
Program  Executive  Officer  (PEO)  for  Littoral  and  Mine  Warfare  (LMW).  PMS  420’s 
responsibilities  originally  involved  end-to-end  LCS  MP  development  and  lifecycle  support.  In 
201 1 ,  PEO  LCS  was  established  to  align  several  program  offices  into  one  consolidated  PEO 
focused  entirely  on  delivering  the  LCS  Program.  This  included  the  establishment  of  PMS 
505  to  assume  the  responsibility  for  Fleet  Introduction  and  Sustainment  of  Mission  Modules 
and  the  seaframes.  PMS  420  continues  to  rely  heavily  on  Participating  Acquisition  Resource 
Managers  (PARMs)  to  deliver  mission  systems  to  meet  LCS  MP  specific  requirements  and 
is  effectively  the  lead  SoS  integrator  of  this  complex  system  of  manned  and  unmanned 
systems.  Examples  of  PARM  systems  include  manned  helicopters,  sensor  systems, 
weapons  systems,  and  unmanned  aerial,  surface,  and  sub-surface  vehicles.  In  some  cases 
where  commercial  or  government  technology  solutions  cannot  be  identified,  PMS  420  works 
with  the  naval  science  and  technology  (S&T)  community  to  conduct  accelerated 
development  of  non-POR  technologies. 
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The  LCS  MMs  are  being  developed  and  delivered  using  a  rapid  fielding  approach 
where  initial  increments  are  fielded  as  the  PARM  technology  matures  and  is  integrated  into 
useful  elements  of  capability.  Key  to  enabling  this  incremental  development  has  been  the 
definition  and  configuration  management  of  standard  interfaces  between  the  seaframe  and 
the  MPs,  as  defined  in  the  Interface  Control  Document  (ICD)  for  the  LCS.  This  approach 
allows  the  seaframes  and  MMs  to  mature  independent  of  each  other’s  development  and 
facilitates  reconfiguration  of  the  LCS  seaframe  and  incremental  upgrade  of  the  modular  MPs 
over  their  life  cycle.  This  independence  of  interfaces,  which  are  managed  by  the  common 
ICD,  allows  both  programs  to  develop  and  test  against  their  unique  KPPs,  key  system 
attributes  (KSAs),  and  milestones.  This  effectively  enables  the  MP  incremental  fielding 
approach.  Useful  increments  of  capability  are  fielded,  which  build  upon  the  initial 
capabilities,  ultimately  satisfying  the  full  KPP  requirements  with  the  final  increment.  The 
incremental  approach  to  delivering  fleet  capability  also  allows  continued  capability  insertion 
throughout  the  life  of  the  program.  Capability  Production  Documents  (CPDs)  are  developed 
to  describe  the  capabilities  required  to  meet  the  production  baseline  for  a  specific  increment 
and  also  provide  a  testable  requirement  for  the  test  community. 

The  LCS  MM  acquisition  strategy  leverages  an  organic  Navy  workforce,  which 
provided  flexibility  to  accommodate  requirements  refinement  as  the  initial  MP  capability 
matured.  This  effectively  reduced  the  Navy’s  exposure  to  cost  and  contract  risk  with  industry 
throughout  the  early  prototyping  phases.  The  various  lead  SoSEs  are  aligned  to  their 
respective  chartered  mission  areas  as  shown  in  Figure  1  (i.e.,  Naval  Surface  Naval  Surface 
Warfare  Center  [NSWC]  Panama  City  Division  is  the  LSoSE  for  the  MCM  MP,  NSWC 
Dahlgren  Division  is  the  LSoSE  for  the  SUW  MP,  Naval  Undersea  Warfare  Center  [NUWC] 
Newport  Division  is  the  LSoSE  for  the  ASW  MP,  while  SPAWAR  Systems  Center  [SSC] 
Pacific  provides  overarching  SoSE  support  for  cross-package  requirements  and  architecture 
activities). 


Figure  1.  LCS  MM  Lead  SoS  Systems  Engineers  by  Mission  Area 

In  a  departure  from  the  traditional  prime  integrator  role,  industry  supports  the  LCS 
MM  program  as  the  Mission  Package  Integrator  (MPI)  to  perform  production,  packaging,  and 
assembly  (PP&A)  functions.  The  MPI  role  is  limited  and  focuses  on  productionizing  systems 
and  equipment  once  a  useful  capability  has  been  demonstrated.  The  various  Naval  Warfare 
and  Systems  Centers  continue  to  assume  primary  responsibility  of  developing  and 
integrating  the  initial  MP  increments  through  initial  testing,  fieldings,  and  early  deployments 
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as  the  LSoSE.  The  LCS  MM  acquisition  workforce  has  also  been  supplemented  significantly 
with  functional  area  subject  matter  expertise  as  available  from  the  various  Navy  Warfare  and 
Systems  Centers  to  provide  cross  MP  support  in  areas  which  include,  but  are  not  limited  to 
the  following:  System  Safety,  Environmental  Safety  and  Occupational  Health,  Information 
Assurance/Cybersecurity,  Configuration  Management,  Human  Systems  Integration, 
Corrosion  Control,  and  so  forth. 

LCS  Mission  Package  and  Open  Architecture 

The  LCS  MMs  program  was  founded  on  the  principles  of  incremental  acquisition, 
open  system  architecture,  and  an  open  business  model.  As  previously  discussed, 
incremental  acquisition  promotes  fielding  of  needed  capabilities  as  they  become  available. 
Standard  interfaces  and  an  open  business  model  enable  more  rapid  integration  of  PARM 
and  other  systems  as  they  mature.  The  LCS  MM  reference  architecture  is  shown  in  Figure 
2.  It  is  a  high-level,  system-agnostic  block  diagram  that  articulates  functional  as  well  as 
business  dimensions.  It  is  not  intended  to  be  the  primary  design  tool,  but  is  used  by  the  lead 
SoSE  and  Program  Manager  (PM)  to  easily  communicate  SoS  boundaries  and  interfaces 
with  internal  and  external  stakeholders,  the  S&T  community,  and  other  stakeholders. 
Additionally,  it  is  used  as  the  basis  to  establish  boundaries  for  configuration  management 
and  certification  events. 

“Focused”  warfare  MPs  are  designed  to  be  installed  on  an  LCS  one  mission  at  a  time 
with  the  LCS  Combat  System  providing  the  MPs  with  access  to  the  seaframe  hosted 
information  technology  (IT)  infrastructure,  sensors,  weapons,  countermeasures,  and 
communications  reachback  through  Exterior  Communications  (EXCOMMS).  Functionality 
provided  by  the  MP  Combat  System  includes,  but  is  not  limited  to  computing, 
communications,  mission  planning,  execution,  and  post-mission  analysis.  MP  manned  and 
unmanned  vehicles  may  be  augmented  with  specific  weapons,  and/or  sensors,  and/or 
countermeasures  to  execute  a  focused  mission.  The  level  of  detail  of  the  reference 
architecture  does  not  allow  for  the  arrows  to  depict  the  specific  data  path,  but  implies 
connectivity  at  the  end  points. 
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Figure  2.  LCS  MMs  Reference  Architecture 


Requirements,  Architecture,  and  Interface  Models 

Orchestrating  the  development  of  a  complex  SoS  requires  clear  communication  and 
organization  of  requirements  in  ways  that  clearly  derive  and  allocate  requirements  as  well  as 
define  boundaries  between  where  responsibility  lies  and  where  negotiation  may  be  required 
between  program  offices,  Systems  Commands,  or  other  stakeholders.  The  LCS  MM  specific 
example  is  further  challenged  by  the  need  to  manage  multiple  increments  at  varying  levels 
of  maturity  and  cost  targets  which  drove  concerted  efforts  to  identify  common  services, 
hardware,  and  systems. 

This  requires  a  robust  and  disciplined  SoSE  activity  to  first  decompose  the  high-level 
requirements  communicated  in  the  CDD  and  CPDs,  and  to  then  perform  the  detailed 
analysis  required  to  support  the  technical  and  cost  trades  needed  to  ensure  that  the 
architecture,  requirements,  and  interfaces  are  developed  and  maintained  at  a  sufficient  level 
of  detail  to  ensure  the  development  of  a  capable  and  cost  effective  SoS  capability.  A  cross- 
MP  Systems  Engineering  Integrated  Product  Team  (SEIPT)  was  established  at  the  working 
level  responsible  for  developing  requirements,  architectures,  and  the  associated  SE  tools 
and  processes  required  to  support  the  development  of  complete,  verifiable,  and  cost 
effective  functional  and  allocated  baselines  for  the  MPs  which  dictate  the  development  of 
the  LCS  MM  SoS.  As  various  increments  proceed  through  the  development,  PARMs  and 
other  external  stakeholders  are  engaged  as  required  to  ensure  that  requirements  and 
interfaces  are  understood  and  managed  effectively.  A  series  of  Technical  Interchange 
Meetings  (TIMs)  are  typically  held  as  a  specific  increment  proceeds  toward  a  formal 
Systems  Engineering  Technical  Review  (SETR;  i.e.,  Preliminary  Design  Review  [PDR], 
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Critical  Design  Review  [CDR],  etc.).  The  LCS  MM  requirements  framework  was  developed 
by  the  SEIPT  and  provides  a  hierarchical  framework  used  to  derive  and  allocate  top-level 
requirements,  prescribed  by  the  CDD,  down  to  the  Mission  Module  and  Mission  System 
levels. 


Level  1:  LCS  Program  Level 


% 


la:  JCIDS 


lb:  Common  Ext  Interfaces 


Level  2:  Commonality  Level 


2a:  Common  Requirements 


Directives 
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Figure  3.  LCS  MMs  Requirements  Framework 

Level  1  requirements  are  defined  as  program  level  requirements  provided  by 
external  sources.  Level  la  requirements  are  developed  by  the  user  through  the  Joint 
Capabilities  Integration  Development  System  (JCIDS)  process.  For  example,  the  LCS  Flight 
0+  CDD  provides  the  required  performance  attributes  for  the  initial  MP  Increments,  and  the 
Incremental  CPDs  provide  the  specific  performance  attributes  in  support  of  the  incremental 
production  and  fielding  of  the  MPs.  Level  1b  requirements  define  common  external 
interfaces  which  are  typically  outside  the  span  of  PMS  420  or  the  LSoSE’s  control.  Level  2 
requirements  are  defined  as  the  common  requirements  developed  by  the  LSoSE.  These 
“common”  or  Level  2  requirements  are  typically  driven  by  open  architecture,  cost  savings,  or 
other  acquisition-type  objectives.  Level  2a  captures  sets  of  like  functional  requirements 
which  are  common  to  more  than  one  Mission  Package  (i.e.,  Safety,  Information  Assurance/ 
Cybersecurity,  Environmental,  Logistics,  etc.).  Level  2b  defines  Systems  and  sub-systems 
which  are  common  to  more  than  one  MP  (i.e.,  computing  infrastructure,  communications, 
sustainment,  etc.).  The  MP  requirements  are  defined  at  Level  3  and  typically  include  the 
level  of  detail  to  understand  the  incremental  nature  of  the  MP  under  development  and/or 
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test.  Mission  Module  requirements  and  intra-MM  interfaces  are  typically  defined  at  Level  4. 
Capabilities  common  to  more  than  one  MM  within  a  specific  MP,  such  as  the  MP  Application 
Software  (MPAS),  are  implemented  at  Level  4.  Mission  System  requirements  and  interfaces 
are  defined  at  Level  5.  Interfaces  which  the  LSoSE  must  retain  control  over  to  ensure 
capability  or  modularity  objectives  are  documented  in  PMS  420  controlled  Interface 
Requirement  Specifications  (IRSs)  or  ICDs. 

The  respective  warfare  focused  MP  technical  teams  manage  the  interfaces  within  an 
MM  or  MP  by  defining  the  details  of  the  interfaces  in  the  requirements  and  architecture 
products.  Internal  MP  interfaces  are  documented  within  the  standard  section  of  the 
System/Sub-System  Specification  (SSS)  and  flowed  down  to  Sub-system  Specification  (SS) 
and  other  documents.  To  control  parallel  development  among  the  various  MPs  or  external 
stakeholders,  PMS  420  defines  and  controls  common  services  and  interfaces  via 
standalone  an  IRS  or  an  ICD/IDD  depending  on  the  level  of  control  required.  An  IRS 
describes  the  functional  requirements  (e.g.,  data  communication  requirements)  and  an  IDD 
or  ICD  describes  the  design  details  of  the  interface. 

Program  offices  responsible  for  constituent  system  development  typically  contract  to 
a  prime  developer  and  find  it  sufficient  to  decompose  the  CDD  into  a  System/Sub-System 
Specification  (SSS)  or  an  A-Spec,  which  is  then  put  on  contract.  Due  to  the  complex  nature 
of  the  LCS  MM  SoS  and  the  number  of  organizations  involved  in  development  of  MMs, 
mission  systems,  sub-systems,  and  so  forth,  PMS  420  has  found  it  necessary  to  manage  its 
specification  tree  down  to  the  MM  level  (or  Level  4)  with  a  strong  understanding  of  how 
those  MM  requirements  are  allocated  to  individual  Mission  Systems.  PMS  420  has 
historically  formalized  such  technical  agreements  with  its  PARMs  or  other  stakeholders 
through  System  Project  Directives  (SPDs),  which  are  essentially  Memorandums  of 
Agreement  between  government  program  offices  supplemented  with  SSS,  Sub-System 
Specification  (SS),  ICDs,  and  architecture  products  as  appropriate.  Architecture  products 
are  typically  developed  through  the  MM  and  mission  system  Levels  4  and  5  in  accordance 
with  version  2.0  of  the  Defense  Architecture  Framework  (DODAF)  framework  (DoN,  2011). 
Interfaces  which  require  high  levels  of  integration  and/or  are  anticipated  to  be  high  risk  are 
further  detailed  using  high  fidelity  interface  models  using  Systems  Modeling  Language 
(SysML).  Interface  requirements  further  derived  through  the  use  of  DODAF  and  SysML  are 
fed  back,  or  synchronized,  into  the  appropriate  requirements  specification  documents.  The 
SPDs  appended  with  SSS,  SS,  and  IRSs  at  Level  4  or  below  to  inform  contract  actions  with 
PARM  contract  actions.  As  most  of  the  mission  systems  which  are  constituent  to  the  SoS 
have  their  own  ACAT  designation,  requirements,  and  funding  lines,  it  is  not  until  this  level  of 
analysis  is  performed  that  we  can  begin  to  understand  the  gaps  between  the  prescribed 
SoS  and  mission  system  requirements.  This  synchronized  requirements  and  architecture 
development  process  is  depicted  in  Figure  4. 

The  requirements  derivation  and  supporting  architecture  development  process 
requires  collaboration  across  several  organizations.  PMS  420  utilizes  a  collaborative 
engineering  environment  such  that  its  distributed  user  base  can  access  the  LCS  MM 
DOORS  modules  from  government  and  contractor  facilities.  In  2014,  this  capability  was 
expanded  to  include  access  to  the  LCS  MM  System  Architect  encyclopedias.  These 
DOORS  and  system  architect  instantiations  are  used  mostly  by  the  PMS  420  core  SE  IPT. 

In  2013,  the  Navy  Systems  Engineering  Resource  Center  (NSERC)  hosted  Systems 
Engineering  Integrated  Data  Environment  (SE  IDE)  was  redesigned  to  ensure  internal  and 
external  users  timely  access  to  requirements,  architectures,  and  technical  data  packages  in 
support  of  formal  reviews  (i.e.,  SETRs,  certification  events,  or  other  events  as  required). 
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Figure  4.  Requirements  and  Architecture  Approach 

Assessing  Technical  Performance  Measures  for  a  Complex  SoS 

Technical  Performance  Measures  (TPMs)  are  a  time-based  metric  used  to  compare 
actual  performance  to  projected  performance  as  a  system  advances  through  various 
developmental  and  test  events.  The  specific  metrics  are  typically  implemented  at  the  system 
or  sub-system  level.  The  TPMs  which  have  the  greatest  impact  on  SoS  mission 
performance  are  specified  in  the  SPD  and  supporting  requirements  documents.  The  PARMs 
are  responsible  to  report  any  scheduled  test  results  back  to  the  LSoSE  on  a  quarterly  basis, 
or  sooner  via  the  integrated  risk  management,  or  other  technical  management  process.  The 
LSoSE  is  then  challenged  with  understanding  the  impact  of  any  performance  shortfalls  and 
then  assessing  the  impact  to  SoS  mission  performance  and  presenting  the  impacts  to 
leadership. 

The  LCS  MM  program  developed  a  framework  to  identify  and  track  the  TPMs  which 
are  critical  for  LCS  mission  success.  Therefore,  thoughtful  selection,  review,  and  tracking  of 
TPMs  are  required  to  allow  technical  managers  to  make  informed  decisions  during  system 
design  and  to  identify  the  need  for  corrective  actions  when  deviations  from  planned 
technical  progress  occur.  In  cases  of  complex  SoS  with  multiple  variables,  dependencies 
and/or  mission  execution  paths,  multiple  TPMs  inform  a  modeling  and  simulation  (M&S)  tool 
to  project  probable  performance  of  the  MSs  and/or  MMs  associated  with  an  MP.  In  some 
cases,  the  TPMs  that  PARMS  are  tracking  for  other  missions  are  not  critical  to  PMS  420  and 
therefore  are  not  tracked  by  the  LCS  MM  LSoSE. 

The  allocation  of  KPPs  to  TPMs  for  a  notional  ASW  MP  is  depicted  in  Figure  5.  High 
fidelity,  non-linear,  complex  M&S  tools  are  usually  required  to  predict  the  performance  of  the 
integrated  MP  based  on  the  TPMs’  inputs  of  the  constituent  mission  systems.  For  example, 
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the  Naval  Mine  Warfare  Simulation  (NMWS)  tool  is  an  object-oriented,  event-driven 
simulation  tool  traditionally  used  by  the  Navy’s  Mine  Warfare  community.  This  SoS  M&S  tool 
is  capable  of  reflecting  the  complexity  of  an  SoS  because  it  can  accommodate  multiple 
inputs,  such  as  mission  thread  information,  environmental  predictions,  mission  system 
performance  parameters,  high-level  reliability  parameters,  and  critical  task  analysis  inputs. 
PMS  420  has  found  the  use  of  a  Kill  Chain  based  modeling  methodology  to  provide  benefit 
to  program  management  and  risk  identification,  and  we  will  now  discuss  that  process. 
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Figure  5.  Notional  TPM  for  a  Notional  ASW  MP 

Applying  the  Kill  Chain  Methodology  to  a  Complex  SoS 

Given  a  notional  SoS  with  12  constituent  systems,  each  with  six  to  eight  TPMs,  over 
100  TPMs  can  result.  The  LSoSE  cannot  always  expect  to  present  status  on  each  TPM  at 
TIMs,  SETRs,  certification  events,  or  other  decision  meetings.  To  assist  in  managing  this 
challenge,  the  Kill  Chain  methodology  of  a  mission-based  approach  to  assessing  and 
visualizing  how  an  SoS  is  expected  to  perform  in  the  execution  of  a  specific  mission  thread 
is  used.  It  is  intended  to  provide  insight  into  any  risks  or  issues  associated  for  the  individual 
systems,  sub-systems,  or  interfaces  in  properly  executing  its  allocated  functionality.  Several 
communities  have  developed  specific  implementations  for  various  purposes.  Perhaps  the 
most  pervasive  are  the  operational  test  community’s  Warfare  Capability  Baseline  (WCB) 
process  and  larger  Navy  Integration  and  Interoperability  (l&l)  implementations  (Clawson  et 
al.,  2015).  The  WCB  approach  in  particular  was  developed  with  a  focus  on  test  data  in 
preparation  for  an  operational  test  event.  LCS  MM  LSoSEs  have  expanded  upon  and 
tailored  a  Kill  Chain  implementation  to  compensate  for  the  incremental,  technical,  and 
organizational  complexities  that  can  complicate  the  assessment  and  presentation  for  a 
complex  SoS. 

As  previously  described,  the  LCS  MM  SoS  is  made  up  of  various  manned  and 
unmanned  vehicles,  sensor  systems,  communication  systems,  weapons  systems,  and 
systems  provided  by  the  core  LCS  seaframe.  As  MPs  are  fielded  incrementally,  mission 
systems  and/or  capability  upgrades  are  added  to  the  MP  baselines.  Further,  multiple  sub- 
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mission  threads,  or  “passes,”  with  various  permutations  of  these  vehicles,  sensors,  and 
systems,  can  be  required  to  accomplish  missions  with  extended  timelines,  such  as  those 
required  to  clear  a  minefield.  An  example  Kill  Chain  for  a  notional  ASW  MP  is  shown  in 
Figure  6.  The  overarching  Kill  Chain  assessment  process  is  depicted  in  Figure  7.  The  Kill 
Chain  is  presented  as  a  matrix  of  systems  and  sub-systems,  laid  against  mission  phases 
(i.e.,  Prepare/Configure,  Search/Detect/Classify/Localize,  Identify,  Neutralize,  Assess,  Post 
Mission  Analysis).  Each  system,  sub-system,  and  integration  point  is  assessed  for  its 
individual  contribution  associated  and  risk  of  meeting  the  mission  specific  requirements  set 
using  the  LCS  MM  standard  risk  assessment  criteria.  In  many  cases,  multiple  TPMs  are 
tracked  and  used  to  quantitatively  assess  the  respective  system  or  sub-system  against  its 
allocated  contributions.  Any  system  which  misses  a  major  TPM  milestone  cues  a  re- 
evaluation  of  the  Kill  Chain  for  impact.  If  the  performance  degradation  does  not  force  a 
change  to  the  risk  rating,  then  the  Kill  Chain  is  not  updated.  The  assessment  method  is  also 
denoted  to  provide  insight  into  the  level  of  confidence  in  the  assessment  (i.e.,  S  =  SME 
estimate,  M  =  Modeled  result,  T  =  Verified  in  test,  F  =  Fielded  System).  A  path  key  with 
varying  thickness  is  used  to  assist  the  PM  or  LSoSE  to  understand  where  the  specific 
system  falls  in  the  multi-pass  sequence  of  events.  Notes  can  be  overlaid  on  the  Kill  Chain  to 
reference  the  PM  or  LSoSE  to  specific  risks  or  issues.  In  order  to  not  lose  the  fidelity,  Kill 
Chains  are  typically  created  for  major  test  configurations,  deployed  configurations,  or 
Increments. 
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Figure  6.  Example  Kill  Chain  for  a  Notional  ASW  MP 
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Figure  7.  LCS  MM  Kill  Chain  Framework 

Lessons  Learned  and  Opportunities  for  Future  Work 

This  paper  presented  the  systems  engineering  processes  and  level  of  technical  rigor 
required  to  manage  the  development  and  integration  of  a  complex  SoS.  A  specific  Kill  Chain 
framework  was  presented,  which  is  currently  used  to  assess  and  visualize  the  impact  of 
TPMs  on  a  complex  systems  of  manned  and  unmanned  systems  mission  threads.  As 
currently  constructed,  the  Kill  Chains  provide  the  PM  with  comprehensive  insight  into  the 
technical  status  and  major  risks  and  issues  for  a  complex  SoS  across  specific  mission 
threads.  Due  to  the  various  configurations  that  the  LCS  MM  PMs  and  LSoSEs  are 
responsible  to  track,  the  LCS  MM  program  office  has  invested  significant  resources  into 
coordinating  several  of  the  basic  systems  engineering  methodologies  and  tools  to  support 
the  Kill  Chain  process,  namely  requirements,  architectures,  and  risk  management. 

•  Requirements  were  developed  to  a  sufficient  level  of  detail  to  articulate  the 
specific  contribution  of  the  system,  sub-system,  and  interface  required  at 
each  phase  in  the  mission  thread. 

•  Architectures  were  developed  to  the  sufficient  level  of  detail  to  ensure  that  the 
weapons  critical  path  covers  the  sequence  of  events  critical  for  mission 
success. 

•  Constituent  system  requirements  must  be  aligned  to  LCS  MM  requirements. 

In  the  cases  when  constituent  systems  have  their  own  requirements  sets 
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(CDDs,  CPDs,  or  other),  the  LSoSE  must  work  with  both  resource  sponsors 
to  identify  and  work  to  close  significant  requirements  and/or  interface 
misalignments. 

•  Constituent  system  program  managers  must  also  be  responsible  to,  and  have 
a  mechanism  to  report,  TPMs  to  the  LSoSE  routinely. 

•  Risk  and  issues  are  flagged  at  least  to  the  level  of  detail  (i.e.,  system,  sub¬ 
system,  interface)  presented  in  the  Kill  Chain. 

•  The  team  responsible  to  develop  MM  requirements  is  distributed  across  at 
least  four  Naval  Warfare  and  Systems  centers.  PMS  420  has  developed  a 
common  access  card  (CAC)-enabled  collaborative  engineering  environment 
where  a  distributed  user  base  can  access  the  LCS  MM  DOORS  modules 
from  government  and  contractor  facilities.  Change  management  and  control 
processes  were  also  developed.  In  2014,  this  capability  was  expanded  to 
include  access  to  the  LCS  MM  System  Architect  encyclopedias. 

To  date,  the  Kill  Chain  framework  has  been  extremely  effective  as  LCS  MM  prepares 
for  several  operational  test  events.  Performance  issues  are  captured,  assessed,  and 
presented  in  the  context  of  their  impact  to  the  overall  mission  success.  In  order  to  expand 
this  approach  to  provide  the  PM  with  a  more  predictive  tool  that  can  be  used  for  technology 
insertion  planning  and  to  better  support  the  developmental  phases  of  the  next  increments, 
some  of  the  other  technical  management  and  systems  engineering  methodologies  and  tools 
must  be  appropriately  configured.  Integrating  a  robust  modeling  and  simulation  capability 
would  better  support  analysis  of  alternative  and  technology  insertion  trade-off-type 
analyses.  Integrating  cost,  schedule,  and  the  -ilities,  such  as  reliability,  sustainability,  and 
other  models,  would  better  support  understanding  the  capability  from  a  performance  and 
total  ownership  cost  (TOC)  perspective. 
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